How does Clojure enforce that there are no side-effects in the transactional code?

Technically it doesn't. If you have a (println "foo") in a transaction it may be printed multiple times if the transaction gets retried.

Clojure does give you an io! macro that you can wrap all your side effects in to manually provide that kind of protection. If you try to use io! inside a transaction you'll get an error at runtime.

    user=> (defn foo [n]
             (io!
               (println n))
             n)
    #'user/foo
    user=> (defn bar [] (dosync (foo 10)))
    #'user/bar
    user=> (bar)
    java.lang.IllegalStateException: I/O in transaction (NO_SOURCE_FILE:0)