
Method setChannelCustomProperty() allows us to specify the channel either by an end point name or the channel name. Take a look at the method in wsadminlib, where there is actually good pydoc to explain the parameters.
The first five parameters are required, in the specified sequence. The last two, endPointName and channelName, are optional and have defaults. You must specify exactly one of them.
So let's add a hypothetical custom property, enablePlayaDust=true, by specifying the channel name. Create a python script file named setChannelCustProp.py:
execfile('wsadminlib.py')
enableDebugMessages()
# Define the parameters.
nodename = 'node1'
servername = 'server1'
propname = 'enablePlayaDust'
propvalue = 'true'
channeltype = 'WebContainerInboundChannel'
channelname = 'WCC_2'
# Set the property
setChannelCustomProperty(nodename, servername, propname, propvalue, channeltype, channelName = channelname)
# Save and sync to all nodes.
save()
Then run it...
root@ding6:/# /opt/WAS70/bin/wsadmin.sh -lang jython -host ding6 -port 8879 -f setChannelCustProp.py
WASX7209I: Connected to process "dmgr" on node dmgr using SOAP connector; The type of process is: DeploymentManager
$Id: wsadminlib.py 50 2009-01-14 14:42:11Z ding $
[2010-0402-1000-0100] enableDebugMessages Verbose trace messages are now enabled; future debug messages will now be printed.
[2010-0402-1000-0100] setChannelCustomProperty: Entry. Setting channel custom property enablePlayaDust=true on node1/server1 channelType=WebContainerInboundChannel endPointName=None channelName=WCC_2
[2010-0402-1000-0200] save: AdminConfig.queryChanges()
[2010-0402-1000-0200] save: WASX7146I: The following configuration files contain unsaved changes:
[2010-0402-1000-0200] save: cells/ndcell/nodes/node1/servers/server1/server.xml
[2010-0402-1000-0200] save: AdminConfig.hasChanges()
[2010-0402-1000-0200] save: AdminConfig.getSaveMode()
[2010-0402-1000-0200] save: rollbackOnConflict
[2010-0402-1000-0200] save: AdminConfig.save()
[2010-0402-1000-0200] save: Save complete!
[2010-0402-1000-0200] wsadminlib.syncall Start
[2010-0402-1000-0300] wsadminlib.syncall Sync config to node node1
[2010-0402-1000-0300] wsadminlib.syncall Done
There were no errors and no exceptions, so that means the channel was found and the prop was set successfully.

PS: Here's a snippet to set a custom prop using the endPointName:
channeltype = 'UDPInboundChannel'
endpointname = 'SIP_DEFAULTHOST'
setChannelCustomProperty(nodename, servername, propname, propvalue, channeltype, endPointName = endpointname)