Is the "Signifier" Arbitrary?

Anyone interested in linguistics will know that for Ferdinand Saussure, the father
of modern structuralism, post-structuralism and semiotics, there existed a "signi-
fier" and "signified": the "signifier" was the linguistic term or phrase used to con-
vey meaning and the "signified" was the object referred to. However, that is not
quite true. Saussure bracketed the referent (or "thing-in-itself") and declared that
the "signifier" referred not to something in the real world, but to a concept in our
minds. This is to say, that for Saussure, the linguistic phrase was representative of
our "idea" of reality--but not necessarily reality as it really is. For example, we
know very well what a corner is--but where does a corner begin and end? Our
perception tells us one thing, but possibly the reality is something else. For these
reasons of verification, Saussure avoided saying that the signified is an object in
the real world: on the contrary, the signified is only a concept in our own minds.
The signifier, for Saussure was "arbitrary", in the sense that there was no neces-
sary connection between the linguistic phrase and the thing or idea represented.
For example, "a house" could just as easily be called "a shoe" as long as everyone
accepted the new meaning ("A rose by any other name would smell as sweet" as
Shakespeare puts it).

In his "Tractatus-Logico-Philosophicus" Ludwig Wittgenstein proposes the idea that

language has a common logic with reality. Admittedly, he eventually renounced
this point of view--but is there any way in which Saussure's theories about epis-
temology could be interpreted as supporting this early Wittgenstinian idea? It
would certainly seem that the theories involved are mutually exclusive, as if lan-
guage is "arbitrary" there can be no common logic between it and reality. How-
ever, the earlier point about Saussure "bracketing the referent" becomes vital
here. If the signifier does not refer to a real thing, but to a concept in our minds,
then it may be possible to assume that though language does not share a common
logic with reality, it does have a common logic with our perception of reality
(which might be constructed from Chomsky's "universal grammar"?). Furthermore,
if our perception of reality portrays a more or less "true reality" then we might
reasonably say that language shares its logic with reality. Of course, the objection
would be: "To what extent does our perception of reality reflect reality as it
really is?" Personally, I believe that all living creatures perceive the world more or
less as it truly is: this is necessary for survival in an often hostile environment.

Was Wittgenstein possibly wrong to discard his early model?