I think you'll still be understood. But the concept of when to use si is not that difficult. It's used whenever you want to say emphatically "yes" to someone who just made a negative statement.
Tu n'es pas prêt. -- Si, je suis prêt.
You're not ready. -- Yes, I am ready.
Elle ne vient plus. -- Si, elle vient encore.
She's not coming anymore. -- Yes, she is still coming.
As you can see, si can be used outside of a question, whereas oui is only possible as a reply when you're asked a question. Oui can also be used when not replying to a negative statement or when agreeing with a negative statement:
Tu es prêt? -- Oui, je suis prêt.
Are you ready? -- Yes, I am ready.
Elle ne vient plus? -- Oui, je crois aussi qu'elle ne vient plus.
She's not coming anymore? -- Yes, I think too that she's not coming.
