challenges and solutions

To discuss on few challenges and their solutions:

Performance problem:
Scenario: Code was initially written in Pro – C. Due to business and technical enhancements, talend tool was used.
Challenge: Validations was done by storing value in variable, which completed occupied the memory heap space. This was done without knowledge of components.
Solution: code was enhanced with help of talend experts and it had a great deal in improving performance
Business situation: Faced this problem in real time. We didn’t face this problem in other testing environments, since input file, we processed was of small size. But in real time (first release), input file was around 100MB, creating havoc among production support team, and was solved in a less than week’s time.

Parallel Processing:
Scenario: slow performance of running jobs (modules) one after another.
Challenge: Even though, performance was improvised with first release, it was taking time for huge files.
Solution: we ran all sub jobs in parallel using a component called tparallelize, leading to a major performance improvement.
Business situation: with enhanced performance, customer was delighted. Now, client has started recommending talend tool for their new projects too.

Latest Trading transaction:
Scenario: We faced a situation in production, where trading was done every second. So in database, we need to delete the antiquated trade details and store the latest one for a particular customer.
Challenge: Multiple threads were running on the talend program. When one thread was deleting and inserting the record, before it committed its transaction, another thread deleted the same record, leading to conflicts. In talend, trading code was complicated and effective use of components was not made. Most code was written directly in java, not giving way to thread synchronization.
Solution: Talend code was modularized i.e. made into smaller modules. A table for locking was introduced in database. Only one thread was allowed to insert/update/delete in all trade tables. Failed transactions of other threads were reprocessed.
Business situation: Criticality of business was very high. Since trading details were not shown properly, customers got pestered. With a limited time, code was developed, tested, deployed.

Efficient logging of Error:
Scenario: Error was logged ineffectively
Challenge: it was difficult to identify the root cause since the same error/warning was logged multiple times.
Solution: we used a component called taggregaterow which groups data based on given data
Business situation: with enhanced logging, issues were resolved instantly and having little dependence on the development team.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: