Bad day at the office

[Dman]

Remembered why we had air pumps at each end of the larval system today; redundancy.
Sometime overnight I lost air to that system and completely lost two of the best hatches we've had to date.
One WSM hatch with 5-600 at meta with 80-90% showing headbands (hatch date 01/01-02) and due to go "on" the sytem today, the other a perc hatch with a little more clowns at the same stage also due "on" the system today.
Needless to say, it wasn't a very friendly atmosphere around here for a couple of hours this afternoon. (not to mention a little smelly)

Just thought I would share this tidbit, build in redundancy where ever you can 'cause you never know when something little like a loose plug will bite ya on the arse.


Thanks for letting me vent

[David M]

Sorry man, hate to hear it

[Heterodonut]

Dman, sorry to hear about your losses. At times like that, it is like a swift kick in the gut. As much as it hurts at the time, I feel times like those are meant to keep us humble. Thanks for the reminder, that these little lives are still fragile, and we need to build our systems in such a way to avoid these "life lessons".

[Kathy55g]

bummer. sorry to hear it. Thanks for the warning.

[fin farm]

Dman,

Sorry to hear about this. I can only imagine what a severe loss like yours feels like. We should take your warning seriously. This can happen to anyone. I think I will get that generator I've talked about getting for the past 2 years.

Larry

[Dman]

So now that yesterday is behind us, we're revisiting all of our systems and processes to re-evaluate when and where redundancy can be built in, yesterday's losses represented a considerable amount of lost possible revenue

[MimicTang]

Where is it good to build redundancy?

[tmays]

Sorry for your loss! Thanks for the wake up call!!!

[aomont]

Dman, sorry to hear... Any rethougth of your system will be valuable to all of us too !
Anderson.

[Kathy55g]

Originally posted by MimicTang
Where is it good to build redundancy?

In anything critical that could fail when you least expect it.

[Dman]

Originally posted by MimicTang
Where is it good to build redundancy?

Well, so far for sure is air.
Our larval system consists of 10 thirty gallon tanks divided in two with an extra 20 (cause there was space). The largest airpump that Hagen makes can run about 40-50 airstones which is more than enough as larval tanks running on the "system" and awaiting transfer to our growout system don't need that much air, if any at all. We had/did add another one of these pumps to the end of the line, therefore if one fo them cuts out the other can handle the load.

The next is water circulation,
Making sure we have doubles of or their equivalent of water pumps to reduce downtime in case of failure. The good thing is that even in the event of water motion failure, most if not all tanks are good for about 48hrs as there's no biofiltration located in any of the tanks. The critical factor with water motion is DO (disolved Oxygen) so what's not covered with the aforementioned paragraph, there's enough water capacity to keep everybody happy.

Temperature is a little more tricky, those of us that aren't David M can relate to real winter temperatures, but we have that covered with an ambient basement temperature of 74F and a gas generator big enough to run a good portion of the shop/basement and house. If we're down for days without access to gas, could be a problem, 6-8 hrs after reaching critical lower temperatures (mid to low 60's) might have me bustin up the furniture.

If there's anything I might have missed, there should be as we didn't get to everything today, please feel free to chime in.

[Peter Schmiedel]

Dman,

sorry to read about the set back!

IMO you also need to consider back up power source in case of general power error. I learned that the hard way loosing my self wild catched group of 7 Centropyge resplendens. Additionally I keep spare parts of all frequently failing components but have to admit that I don't have the membranes for the Hagen pumps ...

[aomont]

Dman will you put to water pumps in one water feeding line ?How do you do this ?
Anderson.

[Dman]

Originally posted by aomont
Dman will you put to water pumps in one water feeding line ?How do you do this ?
Anderson.

I'm not entirely sure this can be done, if one pump is more efficient than the other (even with the same model) there could be backpressure or cavitation issues if their plumbed in sequence. I suppose they could be plumbed in series feeding the same line, but that could present it's own problems that I'm not aware of. Two holes in the sump, two bulkheads and two identical pumps feeding a TEE might work. I'm have to look into that.

Either way water circulation in broodstock or growout systems is nowhere near as critical as air supply in stand alone tanks with hundreds of larvae.

So far we've covered air, water circulation and temperature. What's left? Lighting, not even remotely critical. Foam fractionation again, not critical. Same goes for UV and RDSB's
'm not running any other equipment as of yet, such as Ozone or large DE filters.

[David M]

Certainly air is critical, everything else you can deal with IF you know it's down. On my (very long) list of Iwantits is one of those auto dialers that will call my cell phone if there is a failure.

As for two pumps I see no reason why a simple manifold wouldn't work just fine. Placing a check valve on each one would prevent water from simply being pumped by the good one through the bad one ( simply recirculating in the sump) if there is a failure.

[MimicTang]

I also liked the 2 pump idea. What are you thoughts on UV usage. I hear yes I hear no but what happens if you dont use one cuz many of you seem not to?

As for lifeguard is it making any nasty by products in the water Ed?

[Kathy55g]

MT, I know of no one who does not use a UV on a central system.

[mano]

In my opinion and work experience you should give priority to your brood stock. Larvae can be "replaced". This is especially important if you only have one pair of brood stock, whatever species it may be.

David,

Have you looked into Sensaphone alert systems? We use these on our systems and they do work pretty well. Not super complicated, like a computer linked system we had at the white seabass hatchery, but it does what it needs to do. It actually saved me this last weekend. The seahorse system had dropped to 68 F (from 73) over a couple of days, due to a heater controller failure, plus this winter weather we've been having. The sensaphone called my cell and I went in and replaced it with a backup controller and the seahorses were fine. If I were you I would definitely look into something like this with all that you got going at your "farm".

Christine

[Cuervo]

Take a look at this..
http://www.automatedoutlet.com/produ...0&cat=0&page=1

Especially you guys with the bigger facilities.

It's the heart of a modular automation system. You can get all kinds of different Input/Outpot goodies for it, including different sensors for PH and Temp. Another option is a module that senses current draw, like if you wanted to monitor whether a particular piece of equipment was running or not.

It does go up in cost as you expand the system, but it's kind of cool that it's scalable up to monitoring/controlling over 1000 devices.

It's very programmable, you can make it do some crazy stuff if you really get into it.

It also has a module to call or page you depending on how you program it.. you can even have it turn on an audible alarm.

And no.. I don't work for them. I was looking into a way to fully automate water changes on my tank a while back, and someone suggested that device as the ticket. It would work, but I haven't gotten it done yet, because the holidays got in the way.

[Kmiec123]

That seems very reasonable..What additional cost are there? probes, connectors, etc..All hooked up on a laptop? Looks cool.

[billsreef]

If you have a cable internet connection or DSL that is always up and running, you can leave a PC running with some simple software to monitor things and email your phone if things go out of wack. A simple pressure switch in your air manifold can take of the aeration system, same with the water. As for redundant pumps, it works if each pump has it's own dedicated intake and use check valves on the output before tying the two pumps into your main line.

[Dman]

Like I didn't bookmark that puppy, that looks like a shoe-in for the new facility, thanks for the link, tres cool.

[Cuervo]

Originally posted by Kmiec123
That seems very reasonable..What additional cost are there? probes, connectors, etc..All hooked up on a laptop? Looks cool.

That piece I linked to is just the brain of the system.

Lemme give a general idea of what you can do with it, and I'll throw up some more links to get an idea for pricing. Most aquarium controllers, you're kind of stuck as far as expandability and even adaptability. The drawback is that this is DIY, or you would need to get someone involved who does homeautomation for a living.

First, you need the brain, which I linked above, I didn't really search around alot for pricing, but it was about $165.

Next, you have to decide what you want to do with the system.
I'm going to just make some stuff up.

Scenario:
Call my cell phone in case of critical failure.
Turn on and off the lights on all of the tanks in the system.
Monitor air pressure from primary air pump, and turn on secondary air pump in case of primary failure.
Monitor temp in the broodstock tanks (all connected together).
Monitor temp in the growout tanks. (all connected together).
Monitor temp in the room.
Monitor Broodstock system for water pump failure.
Monitor Growout system for water pump failure.

Alrighty.. easy stuff first.
Temperature - I'll need 3 of these:
http://www.smarthomeusa.com/Shop/PC-...em/TEM-BOBCAT/
They are $60 each - so that's $180 total.

Modem - to be able to call me, it also offers the ability to dial into the system. It's kind of pricey at $169.
http://www.homeautomationnet.com/sho...exd.asp?id=899

I'm going to need two of these:
http://www.smarthomeusa.com/Shop/PC-.../Item/SECU-16/
They each give 8 ports that can be used for input and 8 for output. For a total of 16 inputs and 16 outputs. They are low voltage, low amperage. They each cost $100 so $200 total.

Let's assume that I have 3 sets of lights I want to be able to turn on and off independantly- Broodstock, Larval, Growout. I'll just plug the lights into a powerstrip, and plug the powerstrip into an outlet connected to a relay. The relay allows me to use the low voltage from the controller to turn on and off high voltage devices.
So the lights are going to take 3 of the 16 outputs.

I'm not to worried about whether the lights actually come on or not, because if they fail it's not really critical. If you are concerned though, they offer this light sensor:
http://www.smarthomeusa.com/Shop/PC-...em/LIT-BOBCAT/

To monitor for a water pump failure, I'm just going to use a couple of float switches in the sump of each system. If the pump cuts off, the water drains, and the switch closes. So that's 2 inputs used. I'll connect my water pumps, 2 on each system with completely seperate plumbing. (if I use check valves, the water would never drain and the float switch would never get triggered.) Seperate plumbing is kind of a hassle, but the redundancy is worth it to me. All 4 pumps will be connected to the system via relays, so that's 4 more outputs.

Last, but not least. A pressure switch tied into the air manifold. They run about $20, for an adjustable one that you can set your desired pressure on. It's probably worth noting that this will allow me to not only monitor for a pump failure, but also for a leak in the system, or even if the pump is starting to fail. Kinda cool, anything that causes a drop in the air pressure. It will take up 1 input. Then my two air pumps will be connected via relay to 2 of the outputs.

Alrighty, so we're looking at a total of 9 outputs, and 3 inputs. I dunno, since I'm only using 3 inputs, maybe I'll double up on pressure switches and foat switches, to get some redundancy there, for a total of 6 inputs.

Sum total, for the automation part of things:
Controller: $165
Secu-16 (2): $200
Modem: $169
Temp Monitors (3): $180

Total Cost: $714 plus a bunch of little bits, like float switches, pressure switches, wiring, etc.

I could save $100 by eliminating 1 output, which gets me down to a single SECU-16 instead of two, maybe tie the broodstock and growout lights all together. On the other hand, I'm sure I would eventually use those 7 extra outputs for something. (hmm autofeeder, audible alarm in the facility, phyto lighting, heaters, maybe a button that turns off the pumps for 10 mins at feeding time..)

Then it's a matter of programming it. To give an idea, you can make it do stuff like:
If it senses a pressure drop (pressure switch opens), cut off primary pump, turn on secondary pump, wait 5 mins, if pressure still low, call for help.

or
If temp too high, turn off that tanks heater, wait 5 mins, if temp still increasing, turn off lighting, if temp still increasing, call for help.

I'm no pro on this thing, I got the info from the DIY section on these boards, and I went and did a bunch of research into home automation. There are plenty of aquarium controllers that get up into the price range that I talked about for this setup, but they don't even get close to being able to do everything I talked about on here. When I found that out, I did even more research.

I know $700 is a bit steep for most of us, but for the peeps with bigger operations it's probably a very worthwhile investment.

I've seen other sorts of things that these can be used for - they have optical switches, motion sensors, and even voice activation modules out there.

Hope you guys enjoy!

[Tashier]

Man that royally sucks!

[GreshamH]

Re: DIY PLC controllers

You could do something like this and add the SMS module onto it

http://archive.reefcentral.com/forum...5&pagenumber=1