Note: This won’t work in El Capitan and later unless you disable System Integrity Protection, which you probably don’t want to do.
In a bit of overzealous simplification, the Messages app in OS X 10.9 Mavericks removes the rather basic ability to turn off the sound effect played when sending a message, which has been possible in Messages/iChat as far back as I can remember. In Mavericks, you can choose whether to play all sound effects, and you can choose among the built-in alert sounds for receiving a message, but you can’t disable just the Sent Message “whoop”, as I’ve always done. (I know when I sent a message. I was there.)
The new limitation mirrors iOS, where turning off the sending sound has never been possible. Fortunately, unlike in (non-jailbroken) iOS, there’s an easy workaround. From Terminal, rename the relevant audio file:
sudo mv "/Applications/Messages.app/Contents/Resources/Sent Message.aiff" "/Applications/Messages.app/Contents/Resources/Sent Message.aiff.bak"
Then restart Messages and send messages in blissful silence.
The above command may have to be repeated after OS updates, at least until someone at Apple becomes sufficiently annoyed and restores the ability to turn off the “whoop”. [Update: But not in Yosemite, alas.]
You can similarly change or disable the sound effects for other actions — all of which used to be configurable from the preferences (and could even be customized for specific people from the buddy list) — by replacing or renaming the other .aiff files in that directory:
- Buddy Logging In.aiff
- Buddy Logging Out.aiff
- File Transfer Complete.aiff
- Invitation Accepted.aiff
- Logged In.aiff
- Received Message.aiff
 I imagine it’s possible to have buddy-specific sound effects using the remarkably-still-present “AppleScript handler” functionality, but I haven’t tried it. ↩