Django on Google App Engine: How to Set Up Eclipse for Local Testing

The Django Helper Application

The Google App Engine supports Django, which is a great way to build a UI.  Getting it set up is a bit tricky if you're using a database, though, since Django has some different expectations about how you will interact with a DB.  Fortunately the good people at Google have provided a Django project to help with this.  Their instructions are on  this page.

This post doesn't tell you anything that you can't find elsewhere - I just pulled it from one of my OneNotes pages to share because it is a bit more concise, and it also has a couple of screen caps from Eclipse.

 

Local Django Configuration in Eclipse

To start your app:

• Go to the directory where you want to create your app, and run "django-admin.py startproject mysite" where "mysite" is the name of your app.
• Navigate to the directory where "manage.py" lives for your app.
• Make sure that app-yaml has an app name in it (pretty much anything will work, I think).
• Fire up the local Django app server with "python manage.py runserver".
• To see your app on the local server, go to: http://localhost:8000/.
• To see the local admin console, go to: http://localhost:8000/_ah/admin/.

 

To add a new view / URL in Django (for more details, see the example from the Django tutorial):

1) Add a new function to views.py

 

 2) Add a regex and the function to urls.py

 

 

 

 

That pretty much does it.  I found that this worked well & gave me really good debugging capabilities (the local Django gives excellent information when debugging is turned on - which is by default).

How-to: Setting up Google App Engine for Simple Sites

I wanted to find a simple way to publish a custom website or two for low cost, and settled on Google App Engine as the best current solution. This post focuses on getting an Eclipse environment up and running to debug and publish your Python and other code to Google App Engine.

I initially planned to do the site in Flex, but changed my mind after looking at SEO problems with Flash applications. It does look like there are solutions for getting search engines to index Flash content, but I haven't put in the effort to really explore them yet. My sites are (for the moment) done in plain old HTML with CSS. The only twist is that the Python code strings together several fragments of HTML code to build each page. (I'll talk about that in a later post.)


The Basics
I won't repeat what Nick Berardi already said in his great how-to post. Please check that out for step-by-step instructions on getting your own domain, creating an app engine application, and uploading your content with one command. (You can do this without a domain, but since it's a whopping $10/year there is little reason not to).

Once you have followed these instructions & have an engine application connected to your domain, there are still a couple of steps you need to follow so you can debug the Python portion of the code.

The following is adapted from Google's tutorial here:

1. Install Python 2.6 if you have not already done so. (There is a newer version, but this is what I used.)
   


2. Install pydev eclipse extension for Eclipse through the Eclipse update manager (Help | Software Updates in Eclipse) using this url: http://pydev.sourceforge.net/updates/


3. Tell PyDev where to find the interpreter (create a pydev project and Eclipse will prompt you to do this). I just browsed to the default install location of "Program Files/Python26/python.exe".


4. Install the google app engine SDK if you have not already done so.


5. Create the "helloworld.py" and "app.yaml" files as described in Google's Tutorial.


6. Test the dev server. I kicked off my mini program with this command-line:
   
   • "\Program Files\Google\google_appengine\dev_appserver.py"
     "/Users/Erik/projects/PythonTestBed/src"
   • This runs the dev app server and kicks off the app.yaml script in
     my PythonTestBed/src directory.


7. Fire up a browser with http://localhost:8080. It should show "hello world".
   

Debugging GAE Python in Eclipse
You can run Python programs in two ways (I tried both & they worked):

• By selecting "Python run" in Eclipse.
• By going to the http://localhost:8080 page as above & refreshing it.

I like to view the Python output inside of Eclipse, so I needed to tell PyDev where the api libraries are under Window | Preferences:



The bottom five folders were added with the "New Folder" button. I probably don't need anything other than the appengine, webob and yaml folders for what I'm doing, but I added the others anyway.

I ran into a hitch in calling urlfetch (I needed to parse an RSS stream and feed it into my site as well-behaved HTML. I'll talk about this in a separate post). Because Google uses a proxy to do the actual fetch, it wouldn't run from my local machine until I imported some stubs:

…and then set the stubs up before calling urlfetch in my code…

This will get you up and running. Getting the Python, html, css and javascript pieces uploading and working together will be the subject of a future post.

Stupid Firefox Tricks

I encountered what I think is a bug in the way Firefox handles Javascript. The fix to the problem was simply not to use any Javascript in the body of the page, and I'd be interested to see if anyone else has encountered this.

The Problem
I found what appears to be a thoroughly annoying bug on Firefox, and I'm wondering if anyone else has seen something similar. I suspect that it may have something to do with my configuration, otherwise I should have seen some other mention of it.

I was trying to embed a Feedburner blog summary in a new website, and this worked fine in IE. If you've never tried it, it involves sticking a chunk of Javascript in the middle of your page. Here is a sample (replace all instances of "[...]" with "<...>"):

[html][body]
...my other html...
[script src="http://feeds.feedburner.com/IndependentInDetroit?format=sigpro"]
...
[/body][/html]

The script tag returns:

document.write('[div class="feedburnerFeedBlock" id="IndependentInDetroit2842309"]');
document.write('[p class="feedTitle"]
[a href="http://independentindetroit.blogspot.com/"]Independent in Detroit[/a][/p]');
...(a dozen more lines of javascript-generated html)...
document.write('[/div]');

On the left is the IE screenshot. Notice that the site banner is at the top of the page, as expected. On the right is the Firefox screenshot, where you can see that the banner is not visible. It also says "Loading" which continues indefinitely or until you click the stop button.

Viewing the source shows something bizarre - the page contains only the expanded javascript. All of the other HTML (head, body, etc.) has been stripped off. It appears that Firefox interprets "script tag inside the body" to mean "redirect to a blank page containing only the javascript". Very odd.

The Solution
My fix was to do a urlfetch from Python (this site is hosted on the Google App Engine), then strip off the "document.write(' " and " ');" portions of each line, then stuff the HTML directly into the page. Here is a snippet of the (very simple) code. The second-last line is there because one of the steps (probably "theResult.content", which converts the result object to a String) escaped the single quotes in the string, so they came out looking like " \' ".

...
# Get the feedburner summary, strip out the javascript, then write it.
result =
urlfetch.fetch("http://feeds.feedburner.com/IndependentInDetroit?format=sigpro")
respOut.write(stripJavaScript(result))
...
def stripJavaScript(theResult):
outString = theResult.content.replace("document.write('", "")
outString = outString.replace("');", "")
outString = outString.replace("\\'", "'")
return outString

It worked well, but isn't a very satisfying solution. Has anyone else encountered this? Did I miss something obvious?

I Love JUGs

I attended the Detroit Java User's Group meeting at ePrize in Pleasant Ridge last night.

See the Google Group for details on the organization, and the Yahoo Group for the mailing list. All you need to do to join the user group is add yourself to the mailing list.

The meeting was a good opportunity to connect with my fellow software nerds, and boasted a pretty good mix of different perspectives and skills. Many of the attendees were people who coded mainly in languages other than Java, including Groovy, Scala, Grails and Flex. One of the participants was a physicist working on a 3D visualization and site navigation interface, which struck me as a particularly interesting project.

The common use of other languages raised an interesting point that we discussed briefly - is Java on the way out? The consensus (which is something I heard from the Java Posse podcast some months ago) is that Java-as-a-language may or may not be, but Java-as-a-platform seems to be gaining ground. The evidence for this is fairly solid - many non-Java languages compile to JVM-executable bytecode, allowing access to the rich set of libraries that Java provides. This includes not only Groovy, Scala and Grails mentioned above, but also JRuby, and Jython, all of which have a significant following.

I won't weigh in on which is the "best" language, since the choice of language depends on many factors, not least of which are "will it do what you want?" and "can you get / do you have developers who are productive in it?"

The presentation itself was done by Srini Penchikala, who has written about Java and other topics many times. His chosen topic was Software Architecture, Then and Now. Srini focused on the different choices that architects have when designing a system, and on the advantages of some newer technologies, such as dependency injection, aspect oriented programming and annotations. (He recently blogged on this topic as well. Here is the direct link to his slide deck.)

Most of the meeting was spent in lively discussion, rather than passive listening. It was enjoyable and informative to follow the debate about annotations and object orientation, with several different points of view presented (and argued about).

Overall this was a pleasant experience that left me with a lot of food for thought. I'll be back next month.

Turning to the JUG...

I'm heading out to the Detroit JUG (Java User Group) this evening for a talk by Srini Penchikala on software architectures. It should be interesting, given the topics he has written about on the O'Reilly site.

This is my first visit with the group, so I'll write up my impressions tomorrow.

Links 11/26/2008

Happy holidays!

Here are some resources I have found useful lately...




Ann Arbor Spark
A fairly comprehensive site with many resources for start-ups. This includes everything from an entrepreneur "boot camp" to regular meet & greet events that match start-ups with developers and other talent.

Digital Edge
This is a site devoted to small tech companies. Joe Minock (Joem32 on twitter) runs the site. Per the site, the focus is on "highly scalable web start-ups or web based businesses". It looks like they're just getting started, but I like the idea.

Twitter!
Not local, but it is turning into a pretty good resource for me to find helpful people and advice. The basic Twitter search is useless for finding people that you don't already know, but resources like Twubble and Twellow can help you find a seed group of people to follow. Check out Chris Thomson's blog entry on this for more information. After you have a dozen or so people, consider adding MrTweet - it will give you an analysis of who else you should be following.
(BTW, I'm visual_thinker on Twitter.)

"Practice" as it applies to development

CNN Money had an interesting article on "deliberate practice".   It essentially says that the top performers in any field get there by relentlessly learning to be better at what they do, rather than due to inborn talent, and that "deliberate practice" is not what most people do.

This is not a new idea, but it seems to be a good one.  Ronny Max blogged about it over two years ago, and she cited "Freakanomics" as her source, which was first published in 2005.

Geoff Colvin (the author of the CNN piece) cited eight elements that distinguish it from what most other people consider practice:

1. It is designed specifically to improve performance.
2. It is usually most effective when repeated a lot.
3. Continuous feedback is available.
4. It is mentally demanding.
5. It is difficult.
6. Practitioners set specific goals focused on process (not just outcome).
7. Practitioners "think about their thinking" while practicing.
8. Practitioners evaluate their own performance carefully.

I like the concept for a couple of reasons, not least of which is that it offers hope for improvement in any endeavor.  I disagree with one of the author's central points - that "talent is overrated".  I suspect that you must have a base level of talent for anything to reach the pinnacle of achievement in it, and that practice is how you turn "base talent" into "world class".


But how does this apply to software development?

Good question.  This isn't like practicing a physical skill, and most of the examples in the article were based on sports.  Here are three concepts that might prove useful:

Design the Goal (Items 1 & 6 above)
Think in advance what skill you want to improve with any given programming exercise.  Give yourself the opportunity to practice the skill as part of the goal (i.e., improving the skill is the primary goal), and decide ahead of time how you will measure performance toward the goal.  Some example skills include:

• Retaining and applying knowledge of the toolset (i.e., using a specific set of libraries or a design approach).
• Writing code rapidly without constantly referring to a book or API doc.
• Improving fidelity to requirements.
• Writing thorough tests quickly.

Give Yourself Constant Feedback (Items 3 & 7)
Ask yourself "how am I doing" regularly. Review the goals that you set for yourself to ensure that you're doing / learning what you wanted to do / learn. Also consider your mental state - are you remaining focused on task?  Are you letting yourself  get distracted? Do you really want to attain this goal?

Evaluate Your Performance (Item 8)
As soon as you're done with the process part of the exercise, consider carefully what you should do differently next time. Take some notes!  You want to do it right next time, rather than avoiding the situation entirely.

Productivity Tools

I have spent a lot of time trying to find the "perfect" system for personal productivity.  Here are a few things that I have learned...

Having a system to capture information is even more useful than I thought it would be.  I use something similar to the system described here, which is based on Microsoft OneNote.  In essence, every day (which means nearly every weekday, and some weekends) I create a new page with the date as the title, and then "dump" information into it.  I add checkboxes for to-do items, etc.  I also keep my "daily log" pages in the folder that synchronizes with OneNote on my phone, so today & the past few days of notes (plus a couple of persistent pages) are always on there.

This has worked out great as a to-do list, since OneNote also lets you see all "flagged" items, including those marked with different types of checkboxes.

This leaves out one very important thing, though - keeping information up-to-date in my own brain.  This is something that all of the GTD Gurus specifically tell you not to do.  The idea being that information and to-do items "take up space" in your mind, and prevent you from thinking more creatively.  There is a point to be made there, but I think that Allen and others take the idea too far.

My take (which really isn't wildly different from Allen's "weekly review" process) if that you should review your own notes regularly.  Things that still seem important the next day (or the next week) probably are.  I also don't think that this process should be overly formal.  Just spot-check some things from last month, last week, and yesterday.  One thing I hate is to constanly review "to-do" lists that I can't yet take action on.  Reviewing thoughts, ideas, and opinions of other people and their ideas is another matter, so I try to keep my "to-do" items down to only a handful, and the other information more verbose.

What is a Systems Architect?

I have been re-growing some software development related skills, with an eye toward going from "Java Team Lead" to "Java Architect". What does "architect" mean, though?

Sun offers a certification for a Java platform "Enterprise Architect". Based on the objectives for the exam, they define it based on experience with system design that takes into account flexibility, security and other factors.

This topic has been discussed in a number of other places, including this Javaworld article, which makes the points that most positions billed as "Architect" slots are really "Senior Developer" positions. The author's take is that an architect has breadth rather than depth of knowledge.

This I agree with. To me, a true "architect" is someone who:

• Knows what technologies are available to solve a given design problem.
• Has a handle on "systems thinking" - a way of approaching problems and designing solutions that looks at the whole rather than decomposing the problem and then attacking the smaller issues that come from this decomposition.
• Also has a software development skillset.
  

The first bullet point is something that you can train yourself to do. It entails keeping abreast of what technologies are available, what they're good for, and a realistic assessment of the maturity of each of them.

The "also" in the last bullet point reflects my perspective that an architect is a "developer and..." rather than someone who is really good at software development. I think that you can be a great architect even if you are just a "good" software developer if you have a handle on other skills. For instance, a strong communicator can help move people toward the best technologies by explaining the benefits. Someone who is a really quick study (and hence can understand new technologies easily) will have more options to choose from when designing a solution.

The "systems thinking" point above is the most difficult one to train. If the only problem-solving approach you have is to deconstruct, you probably won't be a great architect (don't lose hope, though - kick-ass software developers are just as valuable, and just as hard to find). Systems thinking is also one of the most difficult things to define for people who don't do it regularly. It is more a way of experiencing the world than it is a discipline. Though there are plenty of descriptions and examples to be found, very few purport to show you how to do it.

So what does this mean for the aspiring architect? Stay in touch with available technologies! Subscribe to blogs and software-related news feeds, join user groups, go to conferences, and read, read, read. If you aren't reading at least one (and preferably two or three) technology book per month, you're falling behind.

Expertise and Competition

Companies are better served by having a few really good people on a project vs. a large number of ineffective ones. This begs a couple of questions:

• Do you potential clients believe that it is better to pay for quality than quantity?
• Who is your competition, and are you better than they are?
  

The Competition

Let's start with the second question: when you are an independent consultant, who is your competition? As a rule, it is not other independent consultants. According to this bureau of labor statistics report, only about 4% of programmers are self-employed. According to this one, only about 6% of systems analysts are self-employed while, this report indicates that only about 2% of "Computer Software Engineers" are self-employed.

There is a lot of overlap between each of these groups, but the bulk of the "tech lead" work will fall into the systems analyst and software engineer groups. That is what I'm mainly seeking, and I assume that you are as well if you're reading this.

So who is the competition? My own experience tells me that when the time comes to staff a project, 90% of the candidates I have seen fit two or more of the following:

• Little or no "initiative" - only do work when given very specific tasks.
• Little or no talent for creating software.
• Non-native speakers of US English with poor communication skills (mainly from India since that's the bulk of the talent pool right now, but many US-born programmers have this problem even without a language barrier).
  
• Limited understanding business in general and the purpose of business software in particular.
• Very little experience with 'real' programming work (as opposed to school projects).

To sum up: if you communicate well and are motivated, you're head-and-shoulders above your competition. (For more on this topic, here is a blog entry with some pretty good quotes on what it takes to be a "good" programmer.)


Your Potential Customers

This is where the rubber really meets the road. I think that most software developers understand on some level what it means to be good or bad in this space. I think that most business people do not know this, but believe that "programmers" are mainly interchangeable.

This state of affairs is changing slowly, and there are a number of voices out there (like this one) that seem to get it.

To find the work you want to do and make a good living at it, you really need to do two things:

1. Sell your clients on the idea that they get a lot more benefit from a few good people (like you!) than from a large group of mediocre ones.
2. Build your expertise to the point where you're obviously one of the best.
   

Both of these will be the subjects of future posts. I would really like to hear your opinions in the meantime!