Yup, that is right. An orange peel, when squeezed to release its oil, can pop a balloon. More specifically, the orange oil contains a nonpolar hydrocarbon called Limonene. The rubber that make up balloons are also nonpolar hydrocarbons.
A molecule is said to be nonpolar when the electron density is evenly spread out, and polar if the distribution is unequal.
A hydrocarbon consists of only hydrogen and carbon atoms.
When nonpolar substances interact they tend to dissolve into each other. In the case of the orange oil and rubber, when the orange oil makes contact with the rubber, the limonene and rubber dissolve into each other thus rupturing the balloon. Pretty wild stuff!
As per usual, The Action Lab has a good video experimenting with this. Other nonpolar liquids work as well, such as regular gasoline.
Short version:
For more information, the following article and videos are a great resource.
Article: https://www.chemedx.org/blog/how-does-orange-peel-pop-balloon-chemistry-course
Orange oil pops balloons:
Some balloons don't pop from orange oil:
The following video shows that not all balloons pop when making contact with orange oil. This is likely due to many balloons being made of vulcanized rubber, which involves adding sulfur with binds and reinforces the hydrocarbon bonds. Water balloons typically aren't made with vulcanized rubber as they are designed to pop easily. Pretty interesting chemistry!