Test Data Generation with Python

My latest task has me writing a utility to generate test data in XML format. My language of choice for this task is Python. At first, I got some push back from management. They told me they thought Python could not handle XML. Ha.

I am starting with the basics. I am using the ElementTree that comes in the Python standard library. It is really easy to parse an XML file in Python. Just takes a few lines of code for ElementTree. The actual parsing is delegated down to a parser program. Expat is such a program that ships with Python.

It is also easy to find or iterate through nodes of interest in the XML. The parsing puts the data in a tree. ElementTree gives you methods to search and destroy. I got hung up for a second because there did not seem to be a method to clone a node. That's okay. I rolled my own.

You need to make sure to do a deepcopy() from the copy module. That way you get the whole node. Then you can append() it to the tree. Make sure to write() the resulting in memory tree back out to an XML file to persist your changes.

Next up I need to figure out which values I need to manipulate in my cloned nodes. Good times I tell you. Good times. No wonder the new guys straight out of college prefer Python for their tools.

Microsoft Virtual Academy

I was planning to complete a Microsoft Virtual Academy (MVA) course on Python. Specifically I was interested in the Developing Web Applications with Python and Flask. Upon review of the topics for the course, I thought it might not be what I really want. I want to write apps. Not so much web apps.

So I perused the MVA courses related to Python. I tried out the Intro to Python. After viewing the first module, I think it might be too rudimentary for me. The presenters were interesting. Their links were out of date. Don't know why. They hosted their stuff on Github. It should still be out there.

Learned that there are a lot of flavors of Python: CPython, JPython, IronPython, PyPy. Not exactly sure what version I have been learning. Maybe these are specific to Microsoft Windows or something. Of course the MVA recommends you use Visual Studio to do your development. So far I have gotten used to the Python Shell that comes with Python 3.4.

One week left until the local Python Hackathon. I think my skills need more practice. Even the MVA guys say you need to practice to learn. Is it time to implement the game of Othello? Or maybe go full blast a try to code up a mini-roguelike?

Python Battleship

I implemented a crude version of the game of Battleship in Python. This is all about x and y coordinates. How should one store such a pair of coordinates? I chose a tuple. A tuple in Python is a set of comma separated numbers. Normally they are programmed within parentheses. I just happened to represent the ship coordinates using tuples. Then I put those tuples in a list.

The good thing about tuples is that you access the elements of a tuple like you would a list, with brackets. A zero based index goes in the bracket. The values in the tuple are immutable. Once set, you cannot change them. This is fine as the ships in Battleship are not changing during the game.

Previously I had learned to use the int() function to convert a string to an integer. Well now I know there is also a str() function to convert an integer to a string. Useful information. Where to go next? I might implement a simple encryption method in Python. Or I might go full tilt and try to code the game of Othello. Still might also want to investigate a Python library. So much to learn. So little time.

Missing From Python

This week I have been concentrating on the features that are present in the Python programming language. However I have noticed some omissions in Python that other programming languages have. One is the case statement. There is no such thing in Python. You got to do a bunch of if else statements.

Another thing that does not seem to be supported out of the box is graphics. Sure you can pipe some characters to the console output. But there are no bitmapped graphics that I can see. Luckily there are popular add on packages that supplement this shortcoming.

I have read about PyGame which will give me graphics and sounds. And there are other choices too I presume. My last big project is going to be implementing the game of battleship. Unfortunately I did not play this game as a kid. I expect that is it something akin to Minesweeper. Let's see how hard that will be using Python.

Python Pass By Reference

I implemented a bare bones version of Tic Tac Toe using the Python programming language. Since my last post, I added logic so that the computer made moves during its turn. The computer AI is not too intelligent. It just follows a preset path, and skips over spots that are taken on the board. The goal here is to learn Python, not figure out the Tic Tac Toe bot.

During this exercise, I needed to finally figure out whether parameters passed to a function are by value or reference. The answer is not so simple. Let's get one thing out of the way first. If you pass a string to a function, it cannot be changed as that type is immutable. But let's discuss passing mutable types to a function.

When you pass a variable, you are supplying a name that is associated to an object. That name is like a reference. The function gets its own reference that is separate from the formal parameter name. However the version that the function has can be used to reference and change the object it points to. So that is kind of like passing by reference.

You can reassign the local function reference to some other object. However the original variable (reference) that the caller has is unchanged by this reassignment. Confused? I guess you got to play with it a bit to understand what the heck is going on here. I will just treat this as pass by reference with some caveats.

Python Tic Tac Toe

I am in the middle of implementing a Tic Tac Toe game using the Python programming language. So far I can draw a board with ASCII graphics. I also get user moves and show them on the board. I can detect whether the latest move is a winning move. Next I need to implement the computer moves. That will require some rudimentary artificial intelligence.

Although I did not have to use it, I figured I should try two dimensional arrays to represent the board. In Python, it would actually be a two dimensional list. The creation of such a beast is not intuitive. I found some weird ways to specify it. However I decided to start simple and create a list with brackets. Then I appended lists to that main list using append(). Then I append the actual moves that sub-list.

The good news is accessing the two dimensional list is just as you would an array: myList[x][y]. I learned a couple other things today. Looks like you have to enclose expressions in parentheses for your if statements. And when there are compound expressions with parentheses themselves, wrap the whole thing in extra parentheses.

I did take advantage of a neat feature in Python. You can return more than one value from a function. Just do a return value1, value2. The caller can use the syntax global1, global2 = fxn(). You don't see that syntax in many (any?) other languages. Now back to tic tac toe implementation.

Python Hangman

I wrote a hangman style game using the Python programming language. This required me to use the skills I learned so far, and stretch a bit to learn more. One thing I picked up was that you can use the keyword elif to act like else if. Saves a few characters.

Also made use of a kind of for loop. You can use for i in range(n) to make i loop from 0 to n-1. Very handy to enumerate the indexes of a string. Speaking of a string, you can use a zero based index in brackets after the string variable name to access individual characters in the string.

The "in" keyword has other uses. You can check whether an item is in a list by using myItem in myList. That is pretty handy. Finally the values for a boolean variable are written True and False. Yes. You must use capital letters to start those keywords.

Now that I got hangman out of the way, it is time to reach for the stars. Next I think I shall implement tic tac toe. That will require a bit of artificial intelligence. But nothing too deep. I might even use the algorithm my book taught me so I can concentrate on the Python programming aspect.

Python Functions

It is pretty easy to write functions in Python. You use the keyword "def". Then you put the name of your function followed by parentheses. Optional argument names go inside the parentheses. After that you place a colon.

The rest of the indented code after the colon is the body of your function. The function can optionally return a value with the return keyword. Now there is variable scope business that I have read about and have not played with. More on that later I presume.

Calls to the function can be made after it has been defined. You just reference the function name followed by parentheses. You put any actual values for the parameters within the parentheses. And you can assign the return value of a function, if any, to the left hand side of the function call.

The colon seems to be standard fare in Python. You put it after the function declaration. You also put if after conditional checks and looping constructs. I am going to be kicking it into high gear next, writing a non-trivial game of hangman in Python.

Python Lists

Time to write my second program in Python. This one will need to hold a bit of data. So I decide to use an array. Ooops. Python does not seem to have an array. But they do have lists. All right. Set a variable name equal to a comma separate list in brackets. Then I have a list.

The list is accessed with a zero-based index in brackets. You can create an empty list with just brackets. Then you can add items using the append() function. The list items do not need to be the same type. But you cannot access an entry in the list that was not created yet. That would cause an error.

I also need some random number generation. I import the random package for help with that. Then I can use the randrange() function which takes the upper and lower bounds of the range for the random integer that is returned.

Next up I am going to learn how to create a function in Python.

Invent With Python

I have been interest in learning the Python programming language for a while. My college had offered a class to teach you Python. But I have not seen the class offered in the last two years or so. Finally I have figured it is time to take matters into my own hands and learn the language by myself.

To help with this, I got a copy of the book Invent Your Own Computer Games with Python. I call it Invent with Python for short. I read through the book over the last month or two. I took a lot of notes. But I did not work through any exercises. Bad move. When I sat down to write my first program, I was clueless.

Now it is time to really learn this language. I did complete my first program with the help of Google. My plan is to write the games that were covered in the book. But I will write them on my own without looking at the source code in the book.

My company is sponsoring a Python hackathon in less than two weeks. Time to really get up to speed so I can complete. All right. I can use the input() function to grab data from the user. That returns a string. Need to convert it to a numeric value using int() before comparing to other integers. I can also use the print() function to do my output.

This is going to be a long ride.

WordPress Carousel

I am taking a college class on WordPress. Have not been really doing any coding in the class. So I am not having a whole lot of fun. I planned out my semester project web site. The initial page was going to have an image slider on it. I figured I would use a Carousel plugin for WordPress. Easier said than done.

I don't know how many plugins I tried. It was at least five. Maybe closer to ten. Lots of them were duds. Yes they were free. But I figured that a carousel was a common item these days. Many times the plugin would not display a single image. Others would display an image or two, but they would not move.

Finally I found Tiny Carousel Horizontal Slider Plus, also known as tchsp. I will admit that I initially had some trouble getting my images into this carousel. Once I did though, the magic happened. Thank you Gopi.

I uploaded a bunch of images to my gallery. However I had trouble getting them to work with tchsp, which requires that you enter the image URL. Turns out if you edit the image in your gallery, there is a File URL in gray over on the right hand side of the screen.

I sure hope the rest of my site will not be as much pain as the carousel was.

Brute Force PHP

I have long since finished my PHP class. Did a simple game earlier this year where I saved the state of the player to my online MySQL database. Now I am teaming up with a friend to write another database application - The Fan Fiction Database.

The minimum viable product is going to only have three fields to display, each of which you can query against. My code is doing a brute force analysis of the combinations of entries the user can make. I have an "if statement" for each combination of inputs. They each produce a custom SQL to execute.

I did do some good stuff in my PHP. It posts back to itself, so I only have one PHP source file. It also detects weird conditions like the database is missing, or there are no matches for your query. I will post the actual user interface in a little while after my partner does her part.

PHP Promotion

I am two weeks into my PHP Programming college course. I am writing small PHP code snippets in my HTML files now. To run the code on my machine, I decided to install WAMP. This gave me automatic installations of Apache, MySQL, PHP, and so on.

To start with, I am just dropping HTML files with PHP code into the root Apache web server. That way I can access them via localhost using my browser. As soon as I started making changes to my PHP code to fix bugs, I found some annoying behavior. I click refresh in Internet Explorer. However the browser just shows the results from the old PHP code.

I eventually found a work around. I could click the Back button in IE, then the Forward button, and finally Refresh would work. That was just too many steps to see the latest output from my new code. I was using Internet Explorer 9. But I doubt that version had anything to do with the problem.

I emailed my instructor. He told me to see him after class. Doh. There he walked me through some settings to jiggle in IE. Specifically I chose Internet Options from the Tools menu in IE. Then I clicked the Settings button under the Browsing History group. There I clicked the radio button that said "Every time I visit the web page" under the "Check for new versions of stored pages" label.

Turns out the default value of "Automatically" does not check for new versions of stored pages. I found this to be a weird location for such an option. But I am just glad that every time I click Refresh, IE actually does refresh my browser with a new run of my PHP code.

JavaScript Library Performance Penalty

Just checked out a deck by Nicholas C Zakas. He shares his insight in making modern web pages load fast. Apparently this is not rocket science. These days a lot of JavaScript library code is downloaded before the page finishes loading. Not all of that code is required up front. You should just download what you absolutely need on page load. Then grab other JavaScript library code on demand. Why didn't I think of that LOL?

Luckily I do not depend on heavy JavaScript libraries in my own code. Heck. I sometimes don't even use jQuery. Yes that is a recipe for pain. But sometimes you got to roll with just your own code. Makes for a quick page load time. You are in full control.

Making JavaScript Fast

The good folks over at Mozilla have posted many tips on how to make JavaScript fast in Firefox. Many of these ideas can speed up JavaScript on any browser. One interesting point was to avoid a lot of local storage calls. Otherwise your main thread gets blocked slowing everything down. That makes sense. Instead you should access a cache of local storage data that is held in memory. Nice.

XSL Plus Local Links

We are coming to the home stretch of my XML college class. The current topic of interest is the eXtensible Stylesheet Language. At first XSL seemed a bit confusing. Like XML Schema, it was not a simple topic if you don't know what the heck is going on. Luckily we got a few homework assignments that are driving the points home.

My latest chore was to produce a list of unique cities from our big file of XML data. No trouble. I put together a little template that compared the current node to the preceeding ones. Any dups got culled, leaving me with a unique list. Simple to put it in order with too. We also needed to show all the raw data, grouped and sorted by city. Another template took care of that in short order.

Next came what seemed to be the tricky part. The unique city listing needed to be comprised of links that took you to the section in the raw data there people from that city are listed. I figured I needed a link to a local in my HTML file. Just need to generate an ID, and reference that ID. And what better a way to generate these anchors? XSL templates of course.

I wonder how some of my other classmates are doing. Some of them got stuck in the XML Schema assignments and never came out. I just hope they know their HTML. It is a requirement for this course.

Almost a Hoodie Ninja

I just read some info on the Hoodie library. This thing claims to let you write web apps quick. One fun thing was that it stores stuff locally, and can work even if you are online. A lot of the sample code looked positively simple. Can this possibly work? I don't know. But it has my interest.

Then I came to the following declaration:

Hoodie is currently a developer preview. Some features are missing, some things might change, there's a lot of optimization to be done. Don't use this for production.

Oh snap. That did not instill confidence. I guess it is nice to get an early look at what they might have going. But I don't fool around much. I write code that I want to ship now. Perhaps I should wait and see how Hoodie pans out. Too bad. It seemed pretty cool.

elementFormDefault

I was working on creating a compound XML Schema document. There was a main xsd file which imported two other xsd files. The problem was in my instance file. I tried to used a prefix for some tags. This worked for tags that were global in the imported xsd files. However the local elements with prefixes did not validate. What the heck?

At first I thought I specified the namespace incorrectly. That lead to nowhere but more problems. I was stumped. When in doubt, do some Googling I always say. In fact, I went back to some example code that I found in a prior Google search. There were some differences between my code and the examples.

I decided to trace down one specific difference. The XML Schema in the examples had an elementFormDefault="qualified" on the top level schema element of the xsd file. As soon as I added that attribute with the qualified value, my problems were solved. I should have known this was related to my issue. Hello? I am trying to qualify local elements with a prefix (to indicate the namespace).

I had seen this attribute before. Just had no idea what it meant. Just goes to show that you need to work through some examples to gain true understanding. I wonder if anyone else in my XML college class is figuring this stuff out?

Compound Schema Document

I went to my XML college class last week. We were talking about using namespaces in XML Schema. The instructor showed me a compoud XML schema document. It referenced multiple namespaces, importing different xsd files. Fair enough. Then he showed me the corresponding instance file. Now this thing was referencing the multiple xsd files as well. WTF?

So I asked why both the instance and main xsd file needed to reference the other xsd files. That did not seem to make much sense. It seemed like overkill. My instructor told me that a lot of things in XML Schema don't make sense. In fact, he told me that it might make more sense if I drank some whiskey. LOL.

Okay. My instructor is good in that he usually backs up class presentations with some homework. I was struggling to get this to work. It needed to display in a browser with some CSS. That part worked fine. Then I needed this to validate in XML Spy. Oh oh. I tried to follow the handouts. But things were just not working.

When I get into trouble like this, I use Google to find web sites to help me dig myself out. I found a great resource from Liquid Technologies. They show how to make a compound schema document. That document references the other XSDs. The instance document only needs to reference the main xsd file. Now that made a whole lot more sense. There was also Definitive XML Schema, although those examples were including other xsd files.

P.S. The funny thing is that I actually own a paper copy of Definitive XML Schema. The book set me back 60 bucks. It is some dry reading. However being able to Google a specific topic from the book seems strong.

Beware JavaScript Math

I am getting heavy into my latest JavaScript project. I am writing a rogue-like game for the web browser. The dungeon is a two dimensional representation. Therefore I am using 2D arrays. That's fine, because JavaScript natively supports them. I am drawing all kinds of ASCII art for the layout of the maze for the dungeon.

For some shapes, I need to do some math to come up with the coordinates. This is nothing fancy. I just iterate through some loops, and do some computations such as add/subtract/multiply/divide. In the middle of development, I found the browser complaining about my computations. What was up?

Some debugging showed me that although my operands were whole numbers, they got converted to a fractional output when I divided. Then I would use the result as an index into an array. That did not turn out too well. Turns out I had to do truncation of the decimal portion using the Math.ceil() function.

There was one other HTML gotcha I encountered late last night. I am using a proportional font to make sure all the cells of the dungeon lined up. However I was having some trouble with the spaces. I thought I would be saved by the proportional font. However I had forgot that the browser ignores the whitespace, trimming batches of spaces down to a single space. I needed to add the non-breaking space entity in there to preserve the spaces.