15 Subscribers
18 Subscriptions
5 Maintainees
 
     
Duncan McGreggor
Valentine, NE View map
63 F - 90 F
Age, tidy epithet,
I'm doing something really interesting.
Would you like to delete the local person associated with this subscription?
Yes No
You already maintain a person with this name. Would you like to:
Use my existing person for this subscription
Or provide a different name for the subscription person:
Invalid person name

Latest
 
09:52 am / 10 hours ago
Weather for Valentine, NE
Current Conditions
59 F
Today
63 F - 90 F
Tomorrow
65 F - 98 F
3 Jul / 22 hours ago
 
3 Jul, 09:10 pm / 22 hours ago
oubiwann: is lovin' him some DragonFly BSD ... a package manager like debian/opensolaris :-) RIP, ports.
oubiwann: is lovin' him some DragonFly BSD ... a package manager like debian/opensolaris :-) RIP, ports.
 
3 Jul, 09:10 pm / 23 hours ago
oubiwann: is lovin' him some DragonFly BSD ... a package manager...
is lovin' him some DragonFly BSD ... a package manager like debian/opensolaris :-) RIP, ports.
 
3 Jul, 08:28 pm / 23 hours ago
oubiwann: Another great Twisted/Divmod blog post: http://ping.fm/w5u8b
oubiwann: Another great Twisted/Divmod blog post: http://ping.fm/w5u8b
 
3 Jul, 08:28 pm / 23 hours ago
oubiwann: Another great Twisted/Divmod blog post: http://ping.fm/w5u8b
Another great Twisted/Divmod blog post: http://ping.fm/w5u8b
 
3 Jul, 08:12 am / A day ago
Welcome to DragonFlyBSD.org
 
3 Jul, 07:22 am / A day ago
Divmod Tech: Making the "Next Gen" Grade
Last night, after I already posted the latest Twisted in the News, I came across another post that would have made the list had I found it sooner. However, this is a good opportunity to give it a little extra attention.

The title of the post is "Next Gen Web Dev: Playing with Python Twisted/Nevow/Athena" and I gotta say, that made my day :-) Between that post and Colin Alston's post that I mentioned in the News, Nevow had a good week. And people are appreciating it for the right reasons. It may not be the easiest web framework to use and certainly not the best documented, but when you need the flexibility to interact with your (Twisted) web server in particular ways as well as benefit from the functionality that COMET provides, Nevow comes out shining.

It's also refreshing to see new developers entering the community who not only see the potential of these tools (designed with that potential in mind) but are capable of taking advantage of it immediately. If nothing else, the author of that post has motivated me to finally merge the Athena tutorial to trunk in order to bring the publicly available and published content in sync with the new code that's in the branch.

Update: Along similar lines, but with more details, Tristan has provided an excellent write-up for this motivation to use Twisted/Nevow/Axiom/Mantissa. Be sure to check it out!

 
3 Jul, 06:41 am / A day ago
oubiwann: Just added weighted load balancing algorithm to txLoadBalancer... getting closer to the next release :-)
oubiwann: Just added weighted load balancing algorithm to txLoadBalancer... getting closer to the next release :-)
 
3 Jul, 06:41 am / A day ago
oubiwann: Just added weighted load balancing algorithm to txLoadBalancer... getting closer...
Just added weighted load balancing algorithm to txLoadBalancer... getting closer to the next release :-)
 
3 Jul, 06:38 am / A day ago
oubiwann: Just added Twitter clone identi.ca to my ping.fm account
oubiwann: Just added Twitter clone identi.ca to my ping.fm account
 
3 Jul, 06:38 am / A day ago
oubiwann: Just added Twitter clone identi.ca to my ping.fm account
Just added Twitter clone identi.ca to my ping.fm account
2 Jul / A day ago
 
2 Jul, 11:27 pm / A day ago
Tv's cobweb: Incremental mapreduce
 
2 Jul, 02:11 am / 2 days ago
SimpleDbIntro - boto - Google Code
 
2 Jul, 02:08 am / 2 days ago
Amazon SimpleDB
1 Jul / 3 days ago
 
1 Jul, 10:02 am / 3 days ago
oubiwann: phew, what a night... heading to bed after hours of hacking on support for a local, in-memory SimpleDB (the txSimpleDB project)
oubiwann: phew, what a night... heading to bed after hours of hacking on support for a local, in-memory SimpleDB (the txSimpleDB project)
 
1 Jul, 10:02 am / 3 days ago
oubiwann: phew, what a night... heading to bed after hours of...
phew, what a night... heading to bed after hours of hacking on support for a local, in-memory SimpleDB (the txSimpleDB project)
 
1 Jul, 07:41 am / 3 days ago
oubiwann: @twleung have you tried Cha-Ching for the Mac?
oubiwann: @twleung have you tried Cha-Ching for the Mac?
 
1 Jul, 07:41 am / 3 days ago
oubiwann: @twleung have you tried Cha-Ching for the Mac?
@twleung have you tried Cha-Ching for the Mac?
 
1 Jul, 02:19 am / 3 days ago
oubiwann: Chatting with Terry Jones about SimpleDB
oubiwann: Chatting with Terry Jones about SimpleDB
27 Jun / 6 days ago
 
27 Jun, 11:03 pm / 6 days ago
oubiwann: @jillianp three cheers for assailing the senses!
oubiwann: @jillianp three cheers for assailing the senses!
 
27 Jun, 11:03 pm / 6 days ago
oubiwann: @jillianp three cheers for assailing the senses!
@jillianp three cheers for assailing the senses!
 
27 Jun, 06:03 pm / A week ago
oubiwann: @jillianp which change would that be?
oubiwann: @jillianp which change would that be?
 
27 Jun, 06:03 pm / A week ago
oubiwann: @jillianp which change would that be?
@jillianp which change would that be?
 
27 Jun, 07:57 am / A week ago
Scrobbled: 94 tracks
By oubiwann
No player available for that track
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
A week ago
 
 
27 Jun, 03:51 am / A week ago
So You Want Your Code to Be Asynchronous? A Twisted Interview
Prologue

This blog post was taken from a chat on a Divmod IRC channel couple weeks ago. Let's start with my opening comments to JP about what I hoped we could accomplish in the interview.

[1:47pm] oubiwann:exarkun: developers/users have started to understand Twisted, see the benefits of an async paradigm, and want to start writing their code making the best possible use of twisted's event driven nature
[1:48pm] oubiwann:they know how to write code using deferreds, and they're ready to get writing...
[1:48pm] oubiwann:except they're not
[1:48pm] oubiwann:because they don't know python internals
[1:49pm] oubiwann:they don't know what python can actually be used with deferreds because they don't know what requirements there are for python code that it be non-blocking in the reactor
[1:50pm] oubiwann:so you're going to help us understand the pitfalls
[1:50pm] oubiwann:how to make best guesses
[1:50pm] oubiwann:and where to look to get definitive answers

Change Your Mind


Before we go any further, I want to share a few comments and answer two questions: "Who is this for?" and "What do I need to know for this to mean something to me?" This post is for anyone who wants to write async code with Twisted and the answer to the second question is open-ended.

Let me start with what is often interpreted as effrontery: read the source code. Despite how that may have sounded, it's not another RTFM quip. The Twisted source code was specifically designed to be read (well, the code from the last two years, anyway). It was designed to be read, re-read, absorbed, pondered, and turned into living memes in your brain.

Understanding tricky topics in conceptually dense fields such as mathematics, physics, and advanced programming requires immersion. When we commit to really learning something difficult in programming, when we take the big step and dive in, we are surrounded by code. At a conceptual level, I mean that literally: it is a spacial experience. This is not something that is typically taught... the lucky few are able to do this their on the own; the rest have to slowly build their intuition through experience in order to get comfortable and be productive in code space.

Our school systems tend to train us along very linear lines: there's a right answer, and a wrong answer. Don't rock the boat. Don't make the teacher uncomfortable. Follow the rules, do your homework, and don't ask too many questions. We carry these habits with us into our professional lives, and it can be quite the task to overcome such a mindset.

Experience is multidimensional. Learning is experience, not rules. When you really jump into this stuff, it will surround you. You will have an experience of the code. For me, that is a mental experience akin to looking at something from the perspective of three dimensions versus two. When I've not dedicated myself to understanding a problem, the domain, or the tools of the domain, everything looks very flat to me. It's hard to muddle through. I feel like I have no depth perception and I get easily frustrated.

When I do take the time, when I make the investment of attention and interest, the problem spaces really do become spaces, ones where my mind has a much greater freedom of movement. It's not smart people who do this kind of thing, it's committed people. Your mind is your world and it's up to you to make it what you want. No one on a mail list or IRC channel can do that for you. They can help you with the rules, provide you with valuable moral support, and guide you along the way. However, a direct experience of the code as a living world of mind comes from taking many brave leaps into the unknown.

Interview in a Blender

Jean-Paul Calderone graciously set aside some time to talk with me about creating asynchronous code in Python, particularly, using the Twisted framework. As has been said many times before, simply using Twisted or deferreds doesn't make your code asynchronous. As with any tricky problem, you have to put some time and thought into what you want to accomplish and how you want to accomplish it.

I'm going to post bits of our chat in different sections, but hopefully in a way that makes sense. There's some good information here and some nice reminders. More than anything, though, this should serve as an encouragement to dig deeper.

Why Would I Ever Need Async Code?

There are a couple short answers to that:
  • Your application is doing many long-running computations (or runs of a varying/unpredictable length).
  • Your application runs in an unpredictable environment (in particular, I'm thinking of network communications).
  • Your application needs to handle lots of events
[1:55pm] oubiwann:exarkun: so, what's the first question a developer should ask themselves as they begin writing their Twisted application/library, txFoo?
[1:55pm] dash:"would everyone be better off if I just stopped now?"
[1:55pm] exarkun:oubiwann: I'm not sure I completely understand the target audience yet
[1:56pm] exarkun:my question is kind of like dash's question
[1:56pm] exarkun:why is this person doing this?
[1:57pm] oubiwann:exarkun: the audience is the group of software developers that are new to twisted, have a basic grasp of deferreds, and want their code to be properly async (using Twisted, of course)
[1:57pm] oubiwann:they don't have anything more than a passing familiarity of the reactor
[1:57pm] oubiwann:they don't know python internals

Protocols, Servers, and Clients, Oh My!

If your application can use what's already in Twisted, you're on easy street :-) If not, you may have to write your own protocols.

Let's get back to the chat:

[1:57pm] exarkun:So `foo´ is... a django-based web application?
[1:58pm] exarkun:... a unit conversion library?
[1:58pm] oubiwann:sure, that works
[1:58pm] oubiwann:unit conversion lib
[1:58pm] oubiwann:(which could be used in Django)
[1:58pm] exarkun:at a first guess, I'd say that there's probably no work to do
[1:58pm] exarkun:how could you have a unit conversion library that's not async?
[1:58pm] exarkun:that'd take some work
[1:59pm] oubiwann:let's say that the unit calculations take a really long time to run
[1:59pm] exarkun:Hm. :)
[1:59pm] idnar:you'd probably have to spawn a new process then :P
[2:00pm] exarkun:basically. probably the only other reasonable thing is for twisted-using code to use the unit conversion api with threads.
[2:00pm] exarkun:so then the question to ask "is my code threadsafe?"
[2:00pm] oubiwann:what about a messaging server
[2:00pm] oubiwann:that sends jobs out to different hosts for calcs
[2:01pm] dash:that's not going to be a tiny example
[2:01pm] exarkun:for that, the job is probably to take all the parsing and app logic and make sure it's separate from the i/o
[2:01pm] exarkun:so "am I using the socket/httplib/urllib/ftplib/XXXlib module?"
[2:03pm] exarkun:is another question for the developer to ask himself
[2:06pm] exarkun:they probably need to find the api in twisted that does what they were using a blocking api for, and switch to it
[2:07pm] exarkun:that might mean implementing a protocol, or it might mean using getPage or something
[2:07pm] exarkun:and pushing the async all the way from the bottom up to the top (maybe not in that direction)
[2:08pm] oubiwann:by "bottom" are you referring to protocol/wire-level stuff?
[2:08pm] oubiwann:exarkun: and by "top" their module's API?
[2:09pm] exarkun:yes
[2:10pm] exarkun:oubiwann: the point being, can't have a sync api implemented in terms of an async one (or at least the means by which to do so are probably beyond the scope of this post)

Processes

We didn't really talk about this one. Idnar mentioned spawning processes briefly, but the discussion never really returned there. I imagine that this is fairly well understood and may not merit as much pondering as such things as threads.

Which brings us to...

Threads

Thread safety is the number one concern when trying to provide an asynchronous API for synchronous code. Here are some starters for background information:
Discussing threads consumed the rest of the interview:

[2:12pm] oubiwann:exarkun: so, back to your comment about "is it threadsafe" (if they are doing long-running python calculations)
[2:13pm] oubiwann:what are the problems we face when we don't ask ourselves this question?
[2:13pm] oubiwann:what happens when we try to run non-threadsafe code in the Twisted reactor?
[2:14pm] exarkun:The problem happens when we try to run non-threadsafe code in a thread to keep it from blocking the reactor thread.
[2:16pm] oubiwann:so non-thread safe code run in deferredToThread could...
[2:16pm] oubiwann:have data inconsistencies which cause non-deterministic bugs?
[2:16pm] dash:have the usual effects of running non-threadsafe code
[2:16pm] exarkun:have any problem that using non-thread safe code in a multithreaded way using any other threading api could have
[2:16pm] dash:like that, yeah
[2:17pm] exarkun:inconsistencies, non-determinism, failure only under load (ie, only after you deploy it), etc
[2:18pm] dash:i smell a research paper
[2:18pm] oubiwann:so, next question: how does one determine that python code is thread safe or not?
[2:19pm] glyph:a research *paper*?
[2:19pm] exarkun:heh
[2:19pm] glyph:research *industry* more like
[2:19pm] oubiwann:exarkun: or, if not determine, at least ask the right sorts of questions to get the developer thinking in the right direction
[2:20pm] dash:glyph: Heh heh.
[2:20pm] exarkun:oubiwann: well, is there shared mutable state? if you're calling `f´ in a thread, does it operate on objects not passed to it as arguments?
[2:20pm] exarkun:oubiwann: if not, then it's probably safe - although don't call it twice at the same time with the same arguments
[2:20pm] exarkun:oubiwann: if so, who knows
[2:20pm] dash:with the same mutable arguments, anyway
[2:23pm] oubiwann:exarkun: so, because python and/or the os doesn't do anything to make file operations atomic, I'm assuming that reading and writing file data is not threadsafe?
[2:24pm] exarkun:don't use the same python file object in multiple threads, yes.
[2:24pm] exarkun:but certain filesystem operations are atomic, and you can manipulate the same file from multiple threads (or processes) if you know what you're doing
[2:25pm] oubiwann:what about C extensions in Python? any general rules there?
[2:25pm] oubiwann:other than "if they're threadsafe, you can use them"
[2:25pm] exarkun:that's about all you can say with certainty
[2:26pm] exarkun:for dbapi2 modules, look at the `threadlevel´ attribute. that's about the most general rule you can express.
[2:26pm] exarkun:there's some stuff other than objects that gets shared between threads too that might be worth mentioning
[2:26pm] exarkun:at least to get people to think about non-object state
[2:27pm] oubiwann:such as?
[2:27pm] exarkun:like, process working directory, or uid/gid
[2:30pm] • oubiwann looks at deferToThread...
[2:31pm] • oubiwann looks at reactor.callInThread
[2:33pm] • oubiwann looks at ReactorBase.threadpool
[2:38pm] oubiwann:hrm
[2:38pm] oubiwann:internesting
[2:39pm] oubiwann:never took the time to trace that all the way back to (and then read) the Python threading module
[2:40pm] oubiwann:exarkun: are there any python modules well known for their lack of threadsafety?
[2:42pm] exarkun:oubiwann: I dunno about "well known"
[2:42pm] exarkun:oubiwann: urllib isn't threadsafe
[2:42pm] exarkun:neither is urllib2
[2:43pm] exarkun:apparently random.gauss is not thread-safe?
[2:43pm] exarkun:you generally start with the assumption that any particular api is not thread-safe
[2:44pm] exarkun:and then maybe you can demonstrate to your own satisfaction that it's thread-safe-enough for your purposes
[2:44pm] exarkun:or you can demonstrate that it isn't
[2:45pm] exarkun:grepping the stdlib for 'thread' and 'safe' is interesting
[2:45pm] oubiwann:I wonder if the stuff available in math is threadsafe....
[2:45pm] oubiwann:exarkun: heh, I was just getting ready to dl the source so I could do that :-)
[2:46pm] exarkun:the math module probably is threadsafe
[2:46pm] exarkun:maybe that's another generalization
[2:46pm] exarkun:stdlib C modules are probably threadsafe
[2:49pm] oubiwann:hrm, looks like part of random isn't threadsafe
[2:51pm] oubiwann:random.random() is safe, though
[2:53pm] oubiwann:exarkun: I really appreciate you taking the time to discuss this
[2:53pm] exarkun:np
[2:53pm] oubiwann:and thanks to dash, glyph, and idnar for contributing to the discussion :-)

Summary

Concurrency is hard. If you want to use threads and you want to do it right and you want to avoid pitfalls and have bug-free code, you're going to be doing some head-banging. If you want to use an asynchronous framework like Twisted, you're going to have to bend your mind in a different way.

No matter what school of thought you follow for any given project, the best results will come with full commitment and immersion. Don't fear the learnin' -- embrace the pain ;-)

Update: Special thanks to Piet Delport for sorting out my endless typos!


26 Jun / A week ago
 
26 Jun, 09:16 pm / A week ago
Pharyngula: Amphioxus and the evolution of the chordate genome

Pharyngula: Amphioxus and the evolution of the chordate genome

This is an amphioxus, a cephalochordate or lancelet. It's been stained to increase contrast; in life, they are pale, almost transparent. It looks rather fish-like, or rather, much like a larval fish, with it's repeated blocks of muscle arranged...

Saved By: oubiwann | View Details | Give Thanks

Tags: , ,

 
26 Jun, 07:15 pm / A week ago
Graphite: Faq
 
26 Jun, 12:36 am / A week ago
Apple vs BMW: Steve Jobs' worst nightmare? Maybe...
A luddite's tale of how Apple isn't so easy to use. Not when you compare it to BMW. Macs would be great if they didn't break so hard and if their "geniuses" actually knew what they were doing when they do. What if BMW got into making computers? Might give Steve Jobs a run for his money...
25 Jun / A week ago
 
25 Jun, 09:30 pm / A week ago
Creating MAC OSX packages for distribution:
 
25 Jun, 09:29 pm / A week ago
CSW - List of packages

CSW - List of packages

Community Software for Solaris.

Saved By: oubiwann | View Details | Give Thanks

Tags: ,

 
25 Jun, 09:23 pm / A week ago
THE ORIGIN OF DOCTOR WHO | A TELETRONIC ARTICLE

THE ORIGIN OF DOCTOR WHO | A TELETRONIC ARTICLE

How the world's longest running science fiction series was conceived.

Saved By: oubiwann | View Details | Give Thanks

Tags: , ,

 
25 Jun, 07:56 pm / A week ago
oubiwann: @glyph SWEEEET!!!
oubiwann: @glyph SWEEEET!!!
 
25 Jun, 07:49 pm / A week ago
Pimp My Safari

 Pimp My Safari

Safari Plugins, Applescripts, Bookmarklets and News

Saved By: oubiwann | View Details | Give Thanks

Tags: , , , , ,

 
25 Jun, 07:34 pm / A week ago
oubiwann: @wsanchez ohnoz! you must refind the fun :-)
oubiwann: @wsanchez ohnoz! you must refind the fun :-)
 
25 Jun, 07:34 pm / A week ago
oubiwann: @wsanchez ohnoz! you must refind the fun :-)
@wsanchez ohnoz! you must refind the fun :-)
 
25 Jun, 04:36 pm / A week ago
Safari 3.1.1 Installer Hosed on OS X 10.5.3
I recently tried updating my Safari to the latest version, only to discover from here and here that Apple seems to have intentionally made this a 10.5.2-only update. I looked in the "Distribution" script and confirmed that this was, in fact, the case. The obvious symptom of this was that the installer told me I couldn't install Safari on any of my drives. Nice.

On those forum posts, I also discovered this great tool: Pacifist. It's been on my backburner list for a while to find a tool that could open up and extract Mac OS X packages, so for that alone I was delighted. When combined with PackageMaker, I was able to create my own installer. Even better.

If this is useful for anyone else, I've put it up here: Safari311UpdLeo_Divmod.pkg. Do note, however, that this installer has no brains: it just puts the files where they should be. It also doesn't check for you system version, so it could potentially really screw things up. Neither I, the Divmod community, nor Divmod, Inc. are responsible in any way if this installer takes your machine to the knacker's yard. However, I am using it on 10.5.3 with no issues (so far).


21 Jun / A week ago
 
21 Jun, 12:22 pm / A week ago
Flickr: 13 images
 
21 Jun, 06:04 am / A week ago
txLoadBalancer
Well today was a flurry of activity... pulled an all-nighter whipping a python load balancer into shape after some late-afternoon discussions on #divmod.

At Divmod, we're going to be labbing out some distributed services experiments with twistd servers, and one set of those experiments involves "developer friendly" load balancing. JP suggested that I take a look at how PyDirector works and see if we could use that. Which was actually interesting in a full-circle kind of way: I worked on PyDirector when I was at PBS, ages ago, where I wrote a weighted lb algorithm for it.

Jumping into the code again after a 5-year hiatus was like seeing an old friend :-)

All tonight, I worked on the following branches:
txLoadBalancer 0.9.1 and 1.0.1 are up on PyPI in the usual place.

I did lots of manual functional testing for each branch tonight, but I didn't do any TDD. While I'm still playing with it, I'll probably start adding tests as bugs crop up (BDT), and as it gets more serious I'll go fully into TDD and fill in what's missing at that point.

Tonight's mad rush was actually a great deal of fun. It's been a while since I've had the opportunity to plow through a bunch of code like that, and I enjoyed myself to near exhaustion :-) I don't think I'll be able to get to sleep tonight (er, this morning), due to the endless thinking about all the ways in which I want to use this code, mutate it, and... well, I better leave some surprises for later!

Update: I've edited the links for the latest micro-releases that fixed some issues with setup.py.

Update 2: Thanks to the heads-up in the comments from Kapil, I've patched txLoadBalancer trunk with the changes from Apple (David Reid and Wilfredo Sanchez).


20 Jun / 2 weeks ago
 
20 Jun, 06:04 pm / 2 weeks ago
A misconception about the ZODB
 
20 Jun, 05:42 pm / 2 weeks ago
Anarchogeek: The Future of Email: From SMTP to XMPP

Anarchogeek: The Future of Email: From SMTP to XMPP

Saved By: oubiwann | View Details | Give Thanks

Tags: