Use wsadminlib's method setCustomPropertyOnObject().
Here is the method signature. The parameters are straightforward:
setCustomPropertyOnObject(object_id, propname, propvalue)
The pydoc is pretty good too. If a property with this name does not exist, the method creates it. If one already exists, the method sets the new value.
Here is an example:
sid = getObjectsOfType('Security')[0]
setCustomPropertyOnObject(sid,'andy.propkey','andy.propvalue')
save()
We can then browse to the admin console and see the new custom property in the list.
Extra info: We can also query the value of a custom property with this method.
getObjectCustomProperty(object_id, propname)