Cyanide is not dangerous in small amounts. Most varieties of bitter nuts have some cyanide in it (ex. almonds)
It only inhibits an enzyme the is necessary for respiration, as long as you don't take enough of it to stop you from breathing, you will be fine. It causes no physical damage.
Cyanide binds irreversibly to the final mediator (cytochrome a3) in your body cells mitochondria (batteries) inhibiting a3 means the cell cannot utilise oxygen - and the cell then has to use anaerobic respiration - without oxygen which results in a build up of lactic acid - and metabolic acidosis which can be fatal if not corrected.
While small amounts might not be immediately fatal - cyanide CAN accumulate if you are subjected to it daily. Your body is only able to detoxify very small quantities - although everyone has some cyanide present i the body !
The LD50 (a lab test - Leathal dose 50 - the dose that will kill 50% of a test population) for mammals is 1 - 3 mg per kg of body weight - but different people react differently.
IF you are at all worried you could do two things -
1) Go get a blood test and get your levels checked !
2) Cyanide has a very high affinity for iron - maybe a filter composed of iron filings or similar might be useful in removing some of it from your drinking water?