Troubleshooting tips for Kinesis streams

The following are common problems and their solutions:

• Unauthorized KMS master key permission error:
  • Without authorized permission on the master key, when a producer or consumer application tries to writes or reads an encrypted stream
  • Provide access permission to an application using Key policies in AWS KMS or IAM policies with AWS KMS
• Sometimes producer becomes writing slower.
  • Service limits exceeded:

Check whether the producer is throwing throughput exceptions from the service, and validate what API operations are being throttled.

 You can also check Amazon Kinesis Streams limits because of different limits based on the call. If calls are not an issue, check you have selected a partition key that allows distributing put operations evenly across all shards, and that you don't have a particular partition key that's bumping into the service limits when the rest are not. This requires you to measure peak throughput and the number of shards in your stream.

• • Producer optimization:

It has either a large producer or small producer. A large producer is running from an EC2 instance or on-premises while a small producer is running from web client, mobile app, or IoT device. Customers can use different strategies for latency. Kinesis Produce Library or multiple threads are useful while write for buffer/micro-batch records, PutRecords for multi-record operation, PutRecord for single-record operation.  

• Shard iterator expires unexpectedly:

The shard iterator expires because its GetRecord methods have not been called for more than 5 minutes, or you have performed a restart of your consumer application.

The shard iterator expires immediately, before you use it. This might indicate that the DynamoDB table used by Kinesis does not have enough capacity to store the data. It might happen if you have a large number of shards. Increase the write capacity assigned to the shard table to solve this.

• Consumer application is reading at a slower rate:

The following are common reasons for read throughput being slower than expected:

• • Total reads for multiple consumer applications exceed per-shard limits. In the Kinesis stream, increase the number of shards.
  • Maximum number of GetRecords per call may have been configured with a low limit value.
  • The logic inside the processRecords call may be taking longer for a number of possible reasons; the logic may be CPU-intensive, bottlenecked on synchronization, or I/O blocking.

We have covered Amazon Kinesis streams. In the next section, we will cover Kinesis Firehose.